Skip to content

feat: upgrade existing filesystem #264

Open
upils wants to merge 22 commits intocanonical:mainfrom
upils:recut
Open

feat: upgrade existing filesystem #264
upils wants to merge 22 commits intocanonical:mainfrom
upils:recut

Conversation

@upils
Copy link
Collaborator

@upils upils commented Jan 29, 2026

  • Have you signed the CLA?

This commit enables Chisel detecting the target directory contains the result
of a previous execution to then operate an upgrade of the content. This initial
simple implementation has the the limitation that content (files, symlinks) is
systematically replaced, even if identical. As a consequences:

  • user modifications on chisel-managed content is overridden.
  • in the context of a OCI image build, this "new" content is identified as
    different and thus the new OCI layer contains duplicated files.

@github-actions
Copy link

github-actions bot commented Jan 29, 2026

Command Mean [s] Min [s] Max [s] Relative
BASE 12.715 ± 0.057 12.613 12.801 1.05 ± 0.01
HEAD 12.068 ± 0.036 12.022 12.115 1.00

@upils upils changed the title feat: update existing filesystem feat: upgrade existing filesystem Feb 2, 2026
@upils upils requested a review from letFunny February 2, 2026 14:35
@upils upils marked this pull request as ready for review February 2, 2026 14:35
Copy link
Collaborator

@letFunny letFunny left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First review. I am missing a couple of files but I wanted to publish it as quickly as possible so we have more time to iterate. This is looking great Paul, many things are looking dead simple which is always an extremely good sign. I have written some comments about how to make the PR shorter and how to make it, hopefully, even simpler. Let me know what you think.

I see some of the comments are similar to past discussions we had in upils#1. We spent some effort there, so please make sure all the comments were carried over to this one.

Comment on lines +169 to +184
fileinfo, err := os.Lstat(path)
if err == nil {
if fileinfo.IsDir() {
if fileinfo.Mode() != o.Mode && o.OverrideMode {
return os.Chmod(path, o.Mode)
}
return nil
}
err = os.Remove(path)
if err != nil {
return err
}
} else if !os.IsNotExist(err) {
return err
}
return err
return os.Mkdir(path, o.Mode)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of things:

  • Prefer early return when possible. In this case it is better to check if os.IsNotExist and fail directly then the rest of the block does not need to be indented.
  • It is now a bit wasteful to do Lstat here followed by another one in fsutil.Create. It would be great if we could unify them.
  • Are we sure we want to remove the path here? This is "breaking compatibility" with the past and it might lead to unexpected deletion in the user side.
  • This feels unrelated to the PR. Maybe as a separate PR (this one is 1400+ lines) and you can state a TODO in this one saying we found a bug in the existing code that will be fixed next.

path := filepath.Clean(filepath.Join(root, relPath))
if !strings.HasPrefix(path, root) {
return "", fmt.Errorf("cannot create path %s outside of root %s", path, root)
return "", fmt.Errorf("cannot handle path %s outside of root %s", path, root)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to change the error message? It feels unrelated to the PR.

}
}

mfest, err := slicer.SelectValidManifest(cmd.RootDir, release)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general our philosophy is that we should be able to cut a release from main at any point in time. That is, if we ever do a bug fix we should be able to release a point version. This is changing how Chisel works in folders with a manifest and it is not the correct behavior (yet) so I would prefer if we could gate this functionality over a environmental variable or a debug command or any other mechanism.

case 0, fs.ModeSymlink:
err = os.Rename(srcPath, dstPath)
case fs.ModeDir:
err = createDir(&CreateOptions{
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not moving the directory, it is left behind. It is also unexpected when an API is called Move and it moves the directory without its contents.

Root: o.DstRoot,
Path: o.Path,
Mode: o.Mode,
OverrideMode: o.OverrideMode,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should always be true or else we are not moving anything, it shouldn't be an option of Move.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point.

summary: "Empty release",
release: &setup.Release{
Packages: map[string]*setup.Package{},
},
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's include expected here to highlight that we should not be getting anything back. As a rule of thumb, every test should always list inputs and outputs that are relevant for the test.

return err
}
if strings.HasSuffix(options.Path, "/") {
err = syscall.Rmdir(path)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good news! The syscall is not actually needed, when I suggested it I had not realized that os.Remove already does the same thing internally, from documentation: Remove removes the named file or (empty) directory.

targetDir = filepath.Join(dir, targetDir)
}

var originalTargetDir string
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A different way of doing it is to extract what used to be the Run function into another one and have Run call into that with different target dir and then upgrade (see extract.go for Run vs extractData). Do you think it will be clearer this way?

}
}

// Remove missing paths
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Remove missing paths
// Remove missing paths.

Comment on lines +413 to +416
err := fsutil.Remove(&fsutil.RemoveOptions{
Root: targetDir,
Path: relPath,
})
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See my other comments in the abstraction and about syscall.Rmdir but I am more convinced now we can call os.Remove directly here. This will also help make the PR shorter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants